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Abstract 

In conjugate gradient method, it is well known that the recursively 
computed residual differs from true one as the iteration proceeds in finite 
arithmetic. Some work have been devoted to analyze this be-havior and 
to evaluate the lower and the upper bounds of the difference. This paper 
focuses on the behavior of these two kinds of residuals, especially their 
lower bounds caused by the loss of trailing digit, respectively. 



1 Introduction 

Conjugate gradient (CG) method and its varieties are popular as one of the best 
unsteady iterative methods for solving the following linear system: 

Ax = b (1) 

In CG method, an approximate solution x% is expected to approach the exact 
solution x*. For the symmetric positive definite A, it is proved that the A- 
norm of the error monotonically decreases as the iteration proceeds in exact 
arithmetic. This will be called as A-norm monotonicity of the error in the 
remaining part of this article. It is obvious that we cannot calculate directly 
such a norm of the error without the solution. Therefore, almost all algorithms 
employ the residual which is easily calculated as the difference between the left 
hand side (LHS) and the right hand side (RHS) of (1), ru := b—Axk- In practice, 
the residual is calculated by the recursion formula because of the computational 
complexity of the matrix vector product Axk [U H]. However, this recursion 
formula causes another problem in which the recursive residual differs from the 
true residual as the iteration proceeds. It can be also observed that the recursive 
residual decreases after the true one seams to reach its lower bound. We should 
terminate the CG steps just before the difference is too large to be neglected. 
Ginsburg has proposed a simple criterion pQ: 

For the true residual calculated as the difference between LHS and 
RHS of a linear system and the recursive residual calculated by using 
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the recursion formula, the procedure is terminated when the 2-norm 
of their difference is greater than the 2-norm of the recursive residual: 

||rjt[| < exp(k/n) 2 \\s k - r k \\ 

where n is dimensionarity of a linear system. 

Several researchers have proposed the estimations of the lower and the upper 
bound of the norm of the error and the residual. Wozniakowski investigated the 
numerical stabilities and good-behaviors of three stationary iterative methods 
and CG method using the true residual b — Ax k O [4]. Wozniakowski gave 
the upper bound of the ultimately attainable accuracies of the A-norm and 
the 2-norm of the error, and 2-norm of the true residual. Bollen gives the 
round-off error analysis of descent methods and lead a general result on the 
attainable accuracy of the approximate solution in finite arithmetic [2] . It has 
also shown that the general result is applied to the Gauss-Southwell method and 
the gradient method to obtain the decreasing rates of the A-norm of the error 
in finite arithmetic. Greenbaum have shown that for tiny perturbation 6m, the 
eigenvalues and the A-norm of the error vectors generated over a fixed number 
of perturbed itera-tive steps are approximately the same as those quantities 
generated by the exact recurrences applied to a " nearby" matrix }5I . The lower 
bound of the true residual is pointed out in [6] . Two kinds of the estimates of 
the A-norm of the error at every step in CG algorithm has been proposed and 
verified that those estimates are the lower and the upper bound in [7]. The 
lower and the upper bounds of the A-norm of the error have been also given 
by Meurant[7] and Strakos and Tichy[8|. Strakos and Tichy have proposed the 
tight estimate for the lower bound of both the A-norm and the 2-norm of the 
error in every step. This stepwise lower bound, however, keeps decreasing after 
the error reaches its 'global' lower bound. Therefore, the terminating criteria 
by using this stepwise lower bound cannot detect the global lower bound of the 
error. Calvetti et al. has proposed the estimates of the lower and upper bound 
of the A-norm of the error in CG method [5]. Those previous studies give the 
stepwise lower and the upper bound of the error and the residual but the global 
bounds. In the remaining part of this article, we will first show that the true and 
the recursive residual almost monotonically de-crease as the iteration proceeds. 
Then, these lower bounds will be shown. 

2 Notation 

We shall give the notations appeared throughout this article. A and b is, re- 
spectively, a coefficient matrix and a constant vector in a linear system. || • || 
in connection with a vector and a matrix, respectively, stands for the 2-norm 
and spectral norm, || • \\a in connection with a vector stands for the norm under 
the metric tensor A. The exact value of a variable x is denoted as x. The 
floating point representation of a variable x is denoted simply as x. The com- 
putational error caused by the floating point representation is denoted as an 
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operator ejvf(x) := x — x. The exact solution of (1) is denoted as x* which is 
described formally as x* = A^ x b. At the k-th step, an approximate solution, 
the error, the true residual, and the recursive residual is, respectively, described 
as Xk, efe, Sfc, and r&. They are computed in CG method as follows: 



Since A is a constant matrix and ||ejvf (A)||/||^4.|| is almost equal to 6m without 
the dependence on the number of iterations, £m(A) is out of our concern as well 
as e M (b). 

3 Almost monotonicity of residuals in finite arith- 
metic 

In this section, we will see the true and the recursive residual has the 2-norm 
almost monotonicity in finite arithmetic, respectively. 

3.1 The 2-norm almost monotonicity of true residual 

The true residual is calculated as the difference between LHS and RHS. This is 
equivalent to multiplication of A to the error in finite arithmetic, 



The behavior of true residual Sk is, therefore, equivalent to Aek- The A-norm 
monotonicity of the error in finite arithmetic has been proved in theorem-3.1 of 
[2] . The following theorem shows the error has the 2-norm almost monotonicity. 

Theorem- 1. If has the A-norm monotonicity, has the 2-norm 
almost monotonicity for a regular matrix A, 



Xk+l 



Tk+l 



Xk + CtkPk, 

* 

X Xk , 

b - Axk, 
r k - a k Ap k 



Sk = b — Axk — b — A(xk — x*) — Ax* 



Ae k 



3k>j,\\e k \\ < ||ej|| 



(2) 



Proof. The relationship between 2-norm and ^4-norm of an error is 



|| efc || = m- 1/2 efc |U<m- 1/2 ]]||e fe |U 



(3) 



similarly, 




(4) 



(5) 
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Equation ([5]) holds if k exists to satisfy the following relation 



||e fc |U< K (A 1 /2)-i||e fe || A 



(G) 



where n(A) is the condition number of a matrix A. From the A- 
norm monotonicity of the error e k , since the following relation holds 
for any positive value a, 



3k > j, \\e k \\ A < a\\e 3 \\ A 



there exists k > j satisties © and consequently ([2]). We have to 
notice that ([2]) does not hold when a = 0, i.e., the inverse of the 
coefficient matrix A is singular. 

Theorem- 1 leads to the 2-norm almost monotonicity of the true residual using 
the relationship ||sfc|| = |j^4efe||. 

Theorem-2. If e k has the 2-norm almost monotonicity, s k has the 
2-norm almost monotonicity for a regular matrix A, 



3k > j, \\s k \\ < \\ Sj \\ 



(7) 



Proof. The relationship between the 2-norm of the error and that 
of the residual is 



|| S)b || = ||A e)b ||<||A||||e fc || 



(8) 



Similarly, 




(9) 



\\A\\\\e k \\<\\A-i\\-' 




(10) 



Equation (jTUJ) holds if k exists to satisfy the following relationship 



||e & |] <«(A)- 1 |]e,-| 



(11) 



From the 2-norm almost monotonicity of the error e k , since the 
following relationship holds for any positive value a, 



3fc > j, \\e k \\ < a\\ej\\ 



(12) 



there exists k > j that satisfies (TTTj) and consequently ([7]). 
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4 The 2-norm almost monotonicity of recursive 
residual 



Before the proof of almost monotonicity of recursive residual in finite arithmetic, 
we first give the proof of almost monotonicity of recursive residual in exact 
arithmetic. From the A-norm monotonicity of the error in exact arithmetic, the 
2-norm almost monotonicity of the residual in exact arithmetic can be proved. 
We have to notice that the recursive residual fj is identical to the true residual 
Sj in exact arithmetic. 

Theorem-3. If Vn, ||e n +i||yi < ||e„||A, then the following proposi- 
toin holds for a regular matrix A: 

3k>j,\\f k \\<\\f j \\ (13) 

Proof. The relationship between the error and the residual gives: 
||r fc || = ||Ae fc || = ||^/ag fc | U (14) 

Then we yield the lower and the upper bound of the 2-norm of the 
true residual: 

p-^II^II^IU^II^II^P^IIIIe.m (15) 



From above equation, the sufficient condition for (|13|) can be given 
as follows: 

|| J 4 1 /2|||| e - fc | U <p- 1 /2||-i|| e - j | U (i 6) 
that is, the equation holds if there exists k > j so that 

||e fe |U< K (yl 1 / 2 )- 1 || e - J |U (17) 

From the A- norm monotonicity of the error e k , the following equa- 
tion holds for any positive value a, 

3k> j,\\e k \\ A <a\\e J \\ A (18) 

and rtTHb holds. 



Now we show the almost monotonicity of the recursive residual in finite arith- 
metic. 

Theorem-4. If the recursive residual has 2-norm almost monotonic- 
ity in exact arithmetic, then the recursive residual has the 2-norm 
almost monotonicity in finite arithmetic: 

3k>j,\\r k \\<\\r j \\ (19) 



5 



Proof. Equation (19) is rewritten as 

3fc > j, \\f k + e M (fk)\\ < \\rj + e M (fj)\\ 

The following relationship is one of its sufficient conditions 

max[||f fe + e M {fk)\\] < min [\\fj + e M (fj)\\] (20) 

The evaluation of the maximum value of LHS is 

max [||rfc||] = (l + e M )|N| (21) 

Similarly, the minimum value of RHS is evaluated as 

min[INI] = (l-e M )IN| (22) 

Substituting and ([22]) into (J20J) , the sufficient condition (20) is 
given as 

3k > j, |jf fc || < (1 - e M )/(l + e M )|NI (23) 

There exists k > j for (1 — €m)/(1 + em) > from theorem-3 and 
((231) holds. 

5 Lower bounds of error and residual in finite 
arithmetic 

It has been shown that the 2-norm of two kinds of residuals, respectively, de- 
creases almost monotonically in finite arithmetic in the previous section. Now 
we consider whether if the 2-norm of each variable stops decreasing before the 
approximates Xk does not reach its target x* . 

5.1 Lower bound of error 

Thcocm-1 shows the approximate solution Xj approaches the exact solution 
almost monotonically in finite arithmetic. The correction of the recursion for- 
mula of Xk, however, vanishes by the loss of trailing digits so that the error stops 
changing, i.e., 

\Ax stop (n)\/\x st0 p{n)\ < e M ==>■ x stop+ i = x stop 

where Xk (n) is the n-th component of Xk ■ 

On the other hand, the solution in finite arithmetic x* is not always identical 
to that in exact arithmetic x* . Therefore, the target for iterative algorithms in 
finite arithmetic should not be x* but x* . The error caused by the loss of trailing 
digits is described formally as x* — x s t op - Since the true residual is given by 
multiplying A to the error, the lower bound of the true residual is given as 
-*4(*£ x s t p*). 
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5.2 Lower bound of recursive residual 

Theorem-4 shows the recursive residual reduces its 2-norm almost monotoni- 
cally. The next theorem proves that the change of the recursive residual stops 
only when ||j"fc+i|| < ejvflkfell' It decreases almost monotonically until then. 

Theorem-5. The recursive residual never have a lower bound caused 
by the loss of trailing digits. 

Proof. The recursion formula of the residual is in general described 
as 

r k+ i = r k - Ak k (24) 

where Akr k '■= a k Ap k . The residual reaches its lower bound r k if 
the following condition is satisfied: 

Ar fe < e M {r k ) (25) 

We will show the condition of (|25|) never be satisfied in not only 
exact but also finite arithmetic. In exact arithmetic, (|24|) satisfies 
the following relationship: 

||r fc+1 || 2 = \\f k ~Af k \\ 2 = \\f k \\ 2 -2(f k ,Af k ) + \\Af k \\ 2 
= ||f fc || 2 -2(f fc ,f fc -f fc+1 ) + ||Af fe || 2 
= -||f fc f + ||Af fe || 2 

where using (fk,fk+\) = 0. The following relationship holds directly 
from above equation in exact arithmetic : 

l|Affe|| > ||ffc|| (26) 

This shows that (|25|) never be satisfied in exact arithmetic. Now we 
evaluate above in finite arithmetic. 



2e M \ \\Arg 
l + e M ) \\r k \\ 

According to (|26|) . we yield 

l|Ar fc || > i 2cm 
Until " l + e M 

and prove that the recursive residual never have a lower bound 
caused by the loss of trailing digits. The termination of the iterations 
is caused only when ||nt + i|| < ejvf||nt|| by the significant decrease of 
the recursive residual for a k Ap k « r k . 



|Ar fc || min [Ar k + e M (Ar k )\ 

\\r k \\ max[||f fc + e A f(^)||] 

(l-e M )||Af fe | 



(1 + tM)\\r k \ 



= 1- 
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6 Conclusions 



In this article, the convergence behaviors of true and recursive residual have 
been analyzed. The results obtained are summarized below: 

• In finite arithmetic, the 2-norm of the error and the residual, respectively, 
almost monotonically decreases. 

• 2-norm of the error has the lower bound in finite arithmetic as well as the 
true residual. 

• 2-norm of the recursive residual never have a non-zero lower bound caused 
by the loss of trailing digits in finite arithmetic. 

References 

[1] T. Ginsburg : "The conjugate gradient method," Numerische Mathematik, 
5(1), pp.191-200 (1963). 

[2] J. A.M. Bollcn : gNumcrical Stability of Descenet Methods for Solving Lin- 
ear Equations, h Numerische Mathematik, 43, pp. 361-377, 1984. 

[3] H. Wozniakowski :gRound-Off Error Analysis of Iteratinos for Large Linear 
Systems,h Numeriche Mathematik, 30, pp. 301-314, 1978. 

[4] H. Wozniakowski :gRoundoff Error Analysis of New Class of Conjugate- 
Gradient Algorithms, h Linear Algebra and its Applications, 29, pp. 507-529, 
1980. 

[5] A. Greenbaum : "Behavior of slightly perturbed Lanczos and conjugate- 
gradient recurrences," Linear Algebra and Its Applications, 113, pp. 7-63 
(1989). 

[6] A. Greenbaum : "Estimating the Attainable Accuracy of Recursively Com- 
puted Residual Methods, h SIAM Journal on Matrix Analysis and Applica- 
tions, 18(3), pp.535-551, 1997. 

[7] G. Meurant, "The computation of bounds for the norm of the error in the 
conjugate gradient algorithm," Numerical Algorithms, 16, pp. 77-87, 1997. 

[8] Z. Strakos, and P. Tichy, "On error estimation in the conjugate gradi- 
ent method and why it works in finite precision computations," Electronic 
Transactions on Numerical Analysis, Vol.13, pp. 56-80, 2002. 

[9] D. Calvetti, S. Morigi, L. Reichel, and F. Sgallari, "Computable error 
bounds and estimates for the conjugate gradient method," Numerical Al- 
gorithms, 25, pp.75-88, 2000. 



8 



